Add Zigbee Green Power (ZGP) protocol support#1814
Draft
Add Zigbee Green Power (ZGP) protocol support#1814
Conversation
Add GP protocol constants (endpoint 242, cluster 0x0021, group ID, default ZigBeeAlliance09 link key) and a comprehensive GPDCommandID enum covering commissioning (0xE0-0xE4), on/off, level control, color control, door lock, and attribute reporting commands as defined in the ZGP specification Table 49.
Implement GP-specific cryptographic operations per ZGP spec A.1.5.4: - build_nonce(): 13-byte CCM nonce from sourceID and frame counter - encrypt/decrypt_security_key(): key protection during commissioning - encrypt/decrypt_payload(): frame payload encryption with variable security levels (NoSecurity, ShortFrameCounter, Full, Encrypted) Uses the cryptography library's AESCCM with 4-byte MIC tags. Includes 27 unit tests covering round-trips, tampering detection, wrong keys/counters, and edge cases.
Implement GPDF commissioning payload parsing per ZGP specification: - GPCommissioningPayload: full parser for commissioning command (0xE0) with options, extended options, security key, key MIC, outgoing counter, and application info (manufacturer/model/commands/clusters) - GPChannelRequestPayload: parser for channel request command (0xE3) - Bidirectional serialization with from_bytes()/to_bytes() Update zgp __init__.py to export crypto, frame, and type modules. Includes 24 unit tests covering all optional field combinations and round-trip serialization.
Implement GPDevice dataclass representing a commissioned GP device:
- sourceID-based identification with synthetic IEEE address generation
(source_id_to_ieee / ieee_to_source_id) matching zigbee2mqtt convention
- Frame counter with replay attack protection
- Security configuration (key, level, key type)
- Device capabilities from commissioning (RX-on, MAC seq, fixed location)
- model_identifier property (GreenPower_{device_id}) for quirks matching
- Full dict serialization/deserialization for persistence
GPDevice intentionally does not subclass zigpy's Device since GPDs have
no ZCL endpoints, binding tables, or standard attribute operations.
Includes 24 unit tests.
Implement the central GP controller (equivalent to zigbee-herdsman's greenPower.ts) that processes GP frames on endpoint 242, cluster 0x0021: - handle_packet(): synchronous entry point from packet_received() - GP Notification processing with device lookup and frame dispatch - Commissioning flow: parse commissioning payload, create GPDevice, decrypt security keys, send GP Pairing to proxies - Decommissioning: remove device, send unpair to proxies - Channel request handling - Commissioning window control via ProxyCommissioningMode broadcast - GP Pairing command construction and broadcast - Device persistence via load_devices()/get_devices_data() - Listener events: gp_device_joined, gp_device_left, gp_command_received Includes 30 unit tests covering packet routing, command dispatch, replay protection, commissioning lifecycle, and persistence.
Hook the GP manager into the application controller: - Instantiate GreenPowerManager as app.green_power in __init__ - Intercept GP packets (ep 242, cluster 0x0021) in packet_received() before the standard device lookup, so GP frames from unknown proxy NWK addresses are handled without triggering device discovery - Add permit_gp(time_s) method separate from standard permit() to avoid accidental GP commissioning during normal Zigbee pairing Includes 7 integration tests verifying GP routing, non-GP passthrough, unknown device handling, and permit_gp delegation.
Implement GPProxyTable to track which GP Proxy devices are forwarding frames for which GPDs from the coordinator (sink) perspective: - add_or_update(): register/update proxy forwarding entries - remove_by_source_id(): cleanup on GPD decommissioning - remove_by_proxy(): cleanup when proxy leaves network - get_proxies_for_device() / get_devices_for_proxy(): topology queries Integrate into GreenPowerManager: - Track proxy NWK on each GP Notification received - Clean up proxy entries on decommissioning Includes 17 unit tests for proxy table CRUD operations.
Comprehensive E2E tests exercising the full GP stack through the real ControllerApplication: - Full lifecycle: commissioning → command dispatch → decommissioning - Commissioning with security key - Commissioning rejection when window closed - GP Notification routing through packet_received() from unknown proxies - Replay protection across sequential and replayed frame counters - Proxy table tracking on notification and cleanup on decommission - Device persistence across save/load cycles 10 tests covering all critical paths.
All bits from 1 to 6 were shifted by +1 due to incorrectly treating MAC Sequence Number Capability as a 2-bit field (bits 0-1) instead of a single bit (bit 0). This caused every subsequent field to be read from the wrong bit position. Corrected layout per Wireshark zbee-nwk-gp dissector and zigbee-herdsman: - Bit 0: MAC Sequence Number Capability (unchanged) - Bit 1: RX On Capability (was bit 2) - Bit 2: Application Information present (was bit 3) - Bit 3: reserved - Bit 4: PAN ID request (was bit 5) - Bit 5: GP Security Key request (was bit 6) - Bit 6: Fixed Location (was bit 7, conflicted with Extended Options) - Bit 7: Extended Options field present (unchanged) Also adds missing test_fixed_location and test_extended_options_present test cases. All test byte values updated to match corrected bit positions.
Per ZGP spec Table 12, SecurityLevel 0b10 (FullFrameCounterAndMIC) provides authentication without encryption: the payload remains in cleartext and only a 4-byte MIC is appended for integrity. Previously, encrypt_payload/decrypt_payload used the same AES-CCM encryption path for all security levels, incorrectly encrypting the payload even for auth-only levels. This would break interoperability with real GPDs using SecurityLevel 0b10. Fix by using CCM* associated data (AAD) mode for auth-only levels: the payload is passed as associated_data with empty plaintext, producing a MIC that authenticates the cleartext without encrypting. Add _is_auth_only() helper to distinguish between encrypted and auth-only security levels. Add test asserting payload identity (output == input) for auth-only and a tamper detection test.
The comments incorrectly documented the bit positions: Direction was listed as bit 2 (should be bit 3, mask 0x08) and Disable Default Response as bit 3 (should be bit 4, mask 0x10). The code was correct, only the comments were wrong. Aligned comments with ZCL spec 2.4.1.1.
Add explicit cleanup for the GP manager, called during ControllerApplication.shutdown(). Cancels any running commissioning window timer and resets the commissioning state, consistent with how OTA, Backups and Topology managers handle their cleanup.
Per the current ZGP specification and all reference implementations (Silicon Labs EMBER_GP_SECURITY_LEVEL_RESERVED, ESP-IDF), SecurityLevel value 0b01 is reserved. The original ZGP 1.0 name "Short Frame Counter and MIC" was deprecated in subsequent revisions. Rename to SecurityLevel.Reserved to align with the current spec and avoid implying this is a functional security level.
The encrypt_security_key function was imported but never used. It would be needed for GP Commissioning Reply to RX-capable GPDs, which is not yet implemented.
Add missing test cases identified during code review: - ApplicationID.LPED enum value - GPDCommandID: identify, scenes, color, door lock, reporting, application description, any command, level control stop - ProxyCommissioningModeExitMode combined values - GPProxyTable.remove_by_proxy with nonexistent proxy - GPCommissioningOptions: fixed_location and extended_options bits
Change struct.pack format from 'b' (signed) to 'B' (unsigned) for the security control byte in build_nonce(). The current value 0x05 works with both, but unsigned is semantically correct for a bitfield and required for future GPP-to-GPD direction support (values >= 0x80).
The ZGP spec requires the GPDF header as CCM* associated data, but it is not available when GP frames arrive via ZCL GP Notification commands. Radio firmware (EZSP, Z-Stack) handles GP decryption natively, making this a non-issue in practice. Same approach as zigbee-herdsman.
Per ZGP spec A.3.6.1.2, the sink must maintain a duplicate filtering table to silently drop GP Notifications forwarded by multiple proxies for the same GPD frame. Without this, each proxy forwarding the same frame triggered a false "possible replay attack" warning. Implement a time-based cache keyed by (sourceID, frameCounter) with a 2-second timeout matching the spec recommendation. Duplicates are logged at DEBUG level instead of WARNING. The frame counter anti-replay check in GPDevice.update_frame_counter() remains as a security measure for genuine replay attacks outside the dedup window. Includes 5 unit tests covering first-pass, duplicate blocking, different sourceID/counter, and cache expiry.
RX-capable GPDs expect a GP Commissioning Reply (cmd 0xF0) containing the security key from the sink. This is not yet implemented. Log a warning so users know why their RX-capable device may not complete commissioning. Most consumer GPDs (EnOcean PTM 215Z, Hue Tap) are not RX-capable and are unaffected.
Add the two missing bit combinations (0b110, 0b111) so all valid 3-bit values are represented. Document that this is semantically a bitmask but uses enum3 due to zigpy's t.Struct serialization constraints (IntFlag is not compatible with bit-level packing).
Add a note to ieee_to_source_id() docstring clarifying that sourceID 0 is reserved/unspecified in the ZGP specification. Validation is left to callers rather than the conversion function itself.
Serialize last_seen as ISO 8601 string in as_dict() and restore it in from_dict(). Previously last_seen was lost on restart, showing "unknown" in the UI until the next GPD event. This improves UX for infrequently-used GP devices like door sensors. Backward compatible: from_dict gracefully handles missing last_seen field (defaults to None).
Fix all ruff violations: - D413: add blank lines after last docstring sections (Google style) - F401: remove unused imports (DeviceID, ApplicationID, FrameType, GPDCommandID in frame.py; source_id_to_ieee, DEFAULT_GP_LINK_KEY in manager.py; DeviceID in device.py) - F841: remove unused variable assignment in test_integration.py - Apply ruff format (line length 88, import ordering, string quotes)
Adjust style to match existing zigpy patterns:
- Use <ClassName ...> angle bracket format for __repr__ methods
instead of ClassName(...) constructor-like format
- Fix GPProxyTable.__repr__ singular/plural ("1 entry" vs "N entries")
- Use tests.async_mock imports instead of unittest.mock directly,
consistent with all other zigpy test files
Document two known differences compared to zigbee-herdsman: - Communication mode is always UnicastLightweight instead of dynamically choosing between Groupcast and Unicast based on the commissioning context - Security key is sent as-is in the Pairing instead of being re-encrypted via encryptSecurityKey() as zigbee-herdsman does Both are acceptable for most home networks but may cause interoperability issues in extended networks or with certain proxy firmware implementations.
Add GP Response (client command 0x06) infrastructure and implement two previously missing GP sink responses: GP Channel Configuration (0xF3): When a GPD sends a Channel Request (0xE3), the sink now responds with the coordinator's operational channel via GP Response. This allows GPDs that scan channels to lock onto the correct one without exhaustive multi-channel commissioning. GP Commissioning Reply (0xF0): When an RX-capable GPD commissions, the sink now sends a minimal Commissioning Reply (options=0x00, no key provisioning) via the forwarding proxy. This matches zigbee-herdsman's approach and allows RX-capable GPDs to complete commissioning without key exchange. Full key provisioning can be added later. Both responses use the new _send_gp_response() helper that constructs a GP Response frame and routes it through the temp master proxy. The proxy_nwk parameter is now propagated from notification handlers through to _process_commissioning and _process_channel_request.
Test coverage for the new GP sink responses: - Channel Config: response sent, correct channel, proxy fallback, invalid payload handling - Commissioning Reply: RX-capable GPD gets reply, non-RX skips it, proxy used as temp master - GP Response: packet structure (ep/cluster/profile), coordinator fallback, send failure handling Also adds network_info.channel to mock fixture for channel-dependent tests.
Add 8 tests with independently computed reference values for AES-CCM operations, replacing sole reliance on round-trip consistency tests. The vectors were generated using raw AESCCM calls with manually constructed nonces (bypassing build_nonce/encrypt_security_key), then cross-validated against the implementation. This ensures the crypto output matches specific expected ciphertext bytes, catching bugs that self-consistent round-trip tests would miss (e.g. wrong nonce structure, wrong CCM mode). Covers: key encryption/decryption, payload encryption/decryption (SecurityLevel.Encrypted), auth-only MIC (FullFrameCounterAndMIC), and nonce byte-level verification.
Fix three tests that validated implementation behavior without checking spec-defined outputs: - test_channel_config_contains_correct_channel: now verifies the GP Channel Configuration byte (channel 20 => offset 9 | basic=1 = 0x19) appears in the sent packet, per ZGP spec - test_commissioning_with_security_key: add assertion on security_key_type (extended byte 0x23 bits 2-4 = NoKey per Table 54), with spec reference in docstring - test_commissioning_reply_uses_proxy_as_temp_master: verify the sent packet is a GP Response (ZCL cmd 0x06) containing GP Commissioning Reply (gpd_cmd 0xF0), check packet count is exactly 2 (reply + pairing)
Fix outgoing_counter=0 being treated as absent due to Python falsy evaluation: - _process_commissioning: use `is not None` check instead of `or` - send_pairing: always include frame_counter in GP Pairing when adding a sink, since frame_counter=0 is a valid initial value per the ZGP spec Add test verifying that outgoing_counter=0 from the commissioning payload is preserved (not replaced by the notification frame_counter).
Cover 5 critical paths that had no test coverage: - Encrypted payload decryption in _dispatch_gp_command: commission a device with SecurityLevel.Encrypted, send an encrypted toggle command, verify the listener receives the decrypted plaintext - Commissioning with encrypted key (key_encrypted=True): encrypt a key with encrypt_security_key, build a commissioning payload with extended options 0x63 (Table 54), verify the device gets the original decrypted key - shutdown() cancels commissioning: open a commissioning window, call shutdown, verify is_commissioning=False and timer cleared - Deduplication within notification flow: deliver the same GP Notification via two different proxies, verify only one gp_command_received event fires - Document that _handle_commissioning_notification (cmd 0x04) cannot be tested due to a pre-existing bit-field alignment issue in the CommissioningNotificationOptions schema (18 bits instead of 16)
Move decrypt_security_key and SECURITY_LEVEL_MIC_LENGTH imports from
inline (inside function bodies) to module-level, consistent with the
existing decrypt_payload import.
Replace the isinstance-based key_mic conversion with a straightforward
struct.pack("<I", comm.key_mic) — the field is always an int when
present (parsed by struct.unpack_from in frame.py).
Improve robustness of the GP manager: - Bound the duplicate filtering cache to 64 entries maximum to prevent unbounded memory growth under heavy GP traffic. When full, the oldest entry is evicted. Also avoid rebuilding the entire dict on each call — check expiry inline and only evict when needed. - Replace bare `except Exception` in handle_packet with specific exception types (ValueError, IndexError, KeyError, AttributeError) to avoid silently swallowing unexpected errors.
The security key field in GP Pairing must be encrypted via encrypt_security_key(sourceID, key) before being sent to proxies, matching zigbee-herdsman's sendPairingCommand() behavior. Proxies decrypt the key using the GP link key to populate their proxy tables. Previously the plaintext key was sent, which could expose it on the Zigbee network and cause interoperability issues with proxies expecting the encrypted form. Add test verifying the plaintext key does NOT appear in the sent packet and the encrypted key DOES.
Choose between UnicastLightweight and GroupcastForwardToDGroup based on whether a specific proxy forwarded the commissioning notification: - proxy_nwk provided: UnicastLightweight with sink IEEE/NWK - proxy_nwk absent: GroupcastForwardToDGroup with GP_GROUP_ID (0x0B84) This matches zigbee-herdsman's sendPairingCommand() which adapts the communication mode based on whether the frame was broadcast or unicast. Decommissioning always uses groupcast (proxy_nwk=None) to notify all proxies to remove the GPD from their tables. Also correctly populates sink_group vs sink_ieee/sink_nwk_addr conditional fields in PairingSchema based on the selected mode.
Per the ZGP spec, sourceID 0x00000000 is reserved as "unspecified" and must not be used by real GPDs. Reject commissioning attempts with this sourceID to prevent creating invalid device entries. Add test verifying the rejection with no device created and no gp_device_joined event fired.
Document the as_dict/from_dict contract in GPDevice: list required and optional keys with their types and defaults, so consumers know what to provide for persistence. Clarify outgoing_counter semantics in GPCommissioningPayload: None means absent, 0 is a valid value (must not be confused with absent). Explain why _build_zcl_frame constructs frames manually: GP frames use a non-standard profile (0xA1E0) and endpoint (242) outside the standard ZCL device/endpoint/cluster lifecycle.
Only ApplicationID.SrcID (0b000) is supported, matching zigbee-herdsman which also ignores IEEE mode. No known consumer GPD uses ApplicationID.IEEE. Supporting it would require changes to the GP cluster schema (greenpower.py) to handle variable-length GPD identifiers, which is outside the scope of the GP manager module.
Replace manual byte-level ZCL parsing in handle_packet with foundation.ZCLHeader.deserialize(), which properly handles frame control bitfields, optional manufacturer ID, TSN, and command ID. Replace manual frame construction in _build_zcl_frame with foundation.ZCLHeader + foundation.FrameControl, ensuring the frame control byte is built using the same typed structs as the rest of zigpy's ZCL stack. Both produce identical bytes to the previous manual implementation but are more maintainable and consistent with zigpy conventions.
Fix three minor issues found during final code review: - frame.py: replace fragile data[offset-1] & 0x80 with options.raw & (1 << EXTENDED_OPTIONS_PRESENT_BIT) for clarity and resilience to refactoring - frame.py: add bounds check on gpd_commands parsing to avoid silently truncated lists when payload data is shorter than num_commands advertises - types.py: add __all__ to control wildcard import exports and prevent internal symbols (like basic) from leaking into the zgp namespace
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dev #1814 +/- ##
==========================================
+ Coverage 99.53% 99.55% +0.01%
==========================================
Files 64 69 +5
Lines 13157 13854 +697
==========================================
+ Hits 13096 13792 +696
- Misses 61 62 +1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
- proxy.py: use max() instead of if/assign per FURB118 - manager.py: fix mypy arg-type error on proxy_nwk by casting packet.src.address to int explicitly - manager.py: fix mypy arg-type on dedup cache min() by using lambda instead of dict.get as key function - manager.py: add noqa: BLE001 to intentional bare Exception catches - manager.py: restructure handle_packet try/except to satisfy TRY300 (move non-exception code out of try block)
Add 22 tests covering previously untested code paths, organized by the ZGP behavior they verify (not implementation details): Sink resilience (spec interop requirement): - Corrupt GP Notification payload silently ignored - Invalid commissioning payload rejected, no device created - Bad encrypted key MIC rejected during commissioning - Corrupted encrypted payload dropped, no event fired Command routing (spec Table 48): - Unknown server command (PairingSearch) ignored - Unexpected client direction ignored - GP Notification routes CommissioningRequest to commissioning - GP Notification routes DecommissioningRequest to removal - GP Notification routes ChannelRequest to channel response - GP SuccessReport accepted silently - Server cmd 0x04 dispatched to commissioning notification handler Dedup cache (spec A.3.6.1.2): - Cache evicts oldest entry when full (DEDUP_MAX_ENTRIES) Network resilience: - ProxyCommissioningMode send failure keeps window open - GP Pairing send failure handled gracefully Timer management: - Re-opening commissioning window cancels previous timer Serialization (spec Tables 53/54/55): - to_bytes with encrypted key + MIC - to_bytes with manufacturer_id and model_id - to_bytes with GPD commands list and cluster list Crypto input validation: - decrypt_security_key rejects bad link_key length - decrypt_payload rejects bad security_key length Mark _handle_commissioning_notification with pragma: no cover due to upstream CommissioningNotificationSchema bit-field bug (18 bits). Coverage: crypto 100%, frame 100%, manager 99% (1 line: asyncio.sleep).
After checking addr_mode == NWK, assert the address is t.NWK so mypy can narrow the union type before the int() cast.
23e0f44 to
96cae53
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add comprehensive Zigbee Green Power (ZGP) protocol support to zigpy, implementing the GP Sink role as defined in the ZGP specification. This is the most requested missing feature (#341, 175+ thumbs-up).
New modules (
zigpy/zgp/)types.pycrypto.pyframe.pydevice.pyproxy.pymanager.pyManager capabilities (equivalent to zigbee-herdsman's
greenPower.ts)Integration (
application.py)packet_received()before device lookup, so frames from unknown proxy NWK addresses don't trigger device discoverypermit_gp(time_s)method separate from standardpermit()green_power.shutdown()called during controller shutdownKnown limitations
ApplicationID.SrcID(0b000) supported — same as zigbee-herdsman; no known consumer GPD uses IEEE modeCommissioningNotificationSchemahas a bit-field alignment issue (18 bits) in the existing GP cluster definition from Green Power Clusters and supporting Schemas #1659 — documented in testsTesting
ruff check+ruff format: 0 violationsTest plan
pytest tests/zgp/— 216 GP tests passpytest tests/test_application.py— 102 existing tests pass (no regression)ruff check zigpy/zgp/ tests/zgp/— 0 violationsResolves #341